Creating Meta-Descriptors of Marketing Messages to Facilitate In Delivery Performance Analysis, Delivery Performance Prediction and Offer Selection

ABSTRACT

Various embodiments are directed to assigning offers to marketing deliveries utilizing new features to describe offers in the marketing deliveries. Marketing deliveries can be described at a finer level to thus enhance the effectiveness of building and conducting marketing campaigns. The approaches facilitate matching content to recipients, predicting content performance, and measuring content performance after dispatching a marketing delivery.

BACKGROUND

A “delivery” is an electronic marketing message, such as an email, SMSor push message, that is sent to many recipients. A delivery typicallyincludes graphics and text, some parts of which are clickable, whileother parts of which are not clickable. Some of the clickable parts areoffers for products or services. By clicking on the clickable parts of adelivery, a consumer can be exposed to details associated with theoffer, such as launching a web page that provides additional productdetails such as available colors, sizes, etc. Deliveries are usuallyprepared by a marketer. When preparing a delivery, a marketer can haveaccess to a catalog of pre-created offers from which offers can beselected. Marketers can select offers by specifying a high level theme,such as “pant suits”, and tags related to the offers, so that onlyrelevant offers are activated (e.g., 25% off pant suits). One problemwith current approach is that offers are not well identified which, inturn, causes the marketer to have to scroll through large numbers ofoffers to find one that is appropriate to his or her marketing message.In addition, offer tags are not automatic and the hierarchy of offersneeds to be manually created. For example, marketers typically have acatalog of assets, such as images, offers and banners, which are indexedin some manner, and can be added into an email. The indexing is donemanually, using tags. Currently tagging of these assets (such as assetshaving bold text or images of humans) is not done automatically.

According to some existing approaches, a marketer may manually createcoded rules to assign offers to recipients. These coded rules maydesignate who gets special offers based on their age, gender, loyaltypoints, the number of deliveries previously sent to them, and the like.Creating these rules is a tedious and cumbersome process. Further, thisprocess does not scale well where the consumers are in the order ofmillions, or when there are many new consumers for whom there is notenough information to enable the consumers to qualify for possibleoffers.

Furthermore, after a message has been dispatched, the marketer willtypically not know which components of the delivery made it successful,nor does the marketer have enough information to replicate thedelivery's success in the future.

SUMMARY

This Summary introduces a selection of concepts in a simplified formthat are further described below in the Detailed Description. As such,this Summary is not intended to identify essential features of theclaimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

Various embodiments are directed to assigning offers to marketingdeliveries utilizing new features to describe offers in the marketingdeliveries. The new features are referred to herein as“meta-descriptors.” The meta-descriptors pertain to how offers arearranged, which products are mentioned, which themes are utilized, andwhich fonts, backgrounds, and color schemes are utilized in connectionwith the delivery. For specific consumers, termed a “target dimension”(such as recipients between age 18-26) important applicablemeta-descriptors are identified which contribute to a campaign's clickrate, conversion rate, and order value per transaction by using apredictive modeling approach. Important applicable meta-descriptors arethose that result in higher key performance indicators than othermeta-descriptors. The predictive modeling approach utilizes aperformance prediction workflow that predicts performance at threedifferent levels—the overall delivery level, the offer level, and themeta-descriptor level. At each level, the predictive modeling approachuses a training module and a testing module.

At the overall delivery level, the training module processes trainingcontent associated with the delivery, such as HTML files associated withthe delivery, along with input provided by a marketer such as keyperformance indicators (KPI) such as open rate, click rate, and thelike, as well as target dimension. The training module extractsmeta-descriptors from the HTML files at the delivery level and processesthe meta-descriptors along with information associated with how thiscontent performed to produce a concatenation that describesmeta-descriptors and content performance for each delivery. Apattern-learning algorithm can be employed to process theconcatenations, i.e., rows of data, to identify which features of thedelivery provide the best performance. The learned model of thesefeatures provides a performance predictor. Performance predictors can bethought of as a set of rules that establish a relationship between thefeatures of the delivery and the actual value of the KPI. This can beperformed by using supervised machine learning.

The testing module then processes test content for the delivery that isto be employed by the marketer. The testing module receives contentassociated with the delivery, such as HTML files associated with thedelivery, along with input provided by the marketer such as KPIs and thetarget dimension to be tested. From this, the testing module extractsmeta-descriptors at the delivery level and processes themeta-descriptors along with information associated with how this contentperformed, e.g., according to a quantification scheme, to produce aconcatenation that describes meta-descriptors and content performancefor each delivery. The testing module then uses the predictor from thetraining module to output a predicted KPI value.

A similar approach can be utilized at the offer level and themeta-descriptor level. The described approaches can be applied topredict the performance of a delivery, offer, and/or meta-descriptors atdesign time, before the delivery is actually produced and sent out toits consumers.

Various workflows can be utilized to enrich offers and deliveries withrelevant tags to help marketers create engaging content.

For example, content of a set of deliveries and meta-descriptorsextracted from the set of deliveries can be received. The content caninclude sets of HTML files associated with the delivery. Marketer inputin the form of KPI and target dimension can also be received. A semanticdescription of the meta-descriptors is created. Next, weights can beassigned to the meta-descriptors according to the performance predictiondescribed above. That is, when the predictors learn whichmeta-descriptors of a particular delivery lead to better KPI values,weights can be assigned accordingly. Thus, better performingmeta-descriptors will receive higher weights. With the weights and thesemantic descriptions having been created, tags associated with thedeliveries can be enriched. The tags are enriched by associating theweights and the semantic descriptions with the individual respectivedeliveries or offers. This creates a searchable collection of deliveriesthat can be searched by a marketer using the enriched tags. In thismanner, marketers can see which deliveries or offers performed betterfor a particular searched tag. This can enable a marketer to select aparticular delivery, which performed better than other deliveries, touse in their marketing campaign.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures.

FIG. 1 is an illustration of a digital medium environment in an exampleimplementation that is operable to employ techniques described herein.

FIG. 2 illustrates an example offer that can be utilized in a delivery.

FIG. 3 illustrates an example delivery building module in accordancewith one or more embodiments.

FIG. 4 is a flow diagram that describes operations in a method inaccordance with one or more embodiments.

FIG. 5 is a flow diagram that describes operations in a method inaccordance with one or more embodiments.

FIG. 6 is a flow diagram that describes operations in a method inaccordance with one or more embodiments.

FIG. 7 is a flow diagram that describes operations in a method inaccordance with one or more embodiments.

FIG. 8 is a flow diagram that describes operations in a method inaccordance with one or more embodiments.

FIG. 9 is a flow diagram that describes operations in a method inaccordance with one or more embodiments.

FIG. 10 is a flow diagram that describes operations in a method inaccordance with one or more embodiments.

FIG. 11 is a table in accordance with one embodiment.

FIG. 12 is a table in accordance with one embodiment.

FIG. 13 illustrates two offers in accordance with one embodiment.

FIG. 14 is a histogram in accordance with one or more embodiments.

FIG. 15 illustrates an example system including various components of anexample device that can be employed for one or more searchimplementations described herein.

DETAILED DESCRIPTION Overview

Various embodiments are directed to assigning offers to marketingdeliveries utilizing new features to describe offers in the marketingdeliveries. The new features are referred to herein as“meta-descriptors.” The meta-descriptors pertain to how offers arearranged, which products are mentioned, which themes are utilized, andwhich fonts, backgrounds, and color schemes are utilized in connectionwith the delivery. For specific consumers, termed a “target dimension”(such as recipients between age 18-26) important applicablemeta-descriptors are identified which contribute to a campaign's clickrate, conversion rate, and order value per transaction by using apredictive modeling approach. Important applicable meta-descriptors arethose that result in higher key performance indicators than othermeta-descriptors. The predictive modeling approach utilizes aperformance prediction workflow that predicts performance at threedifferent levels—the overall delivery level, the offer level, and themeta-descriptor level. At each level, the predictive modeling approachuses a training module and a testing module.

At the overall delivery level, the training module processes trainingcontent associated with the delivery, such as HTML files associated withthe delivery, along with input provided by a marketer such as keyperformance indicators (KPI) such as open rate, click rate, and thelike, as well as target dimension. The training module extractsmeta-descriptors at the delivery level and processes themeta-descriptors along with information associated with how this contentperformed to produce a concatenation that describes meta-descriptors andcontent performance for each delivery. Content performance can bequantified using any suitable quantification scheme, e.g., on a scalebetween 1-100, with 1 being the worst and 100 being the best. Eachconcatenation consists of one row of data which includes themeta-descriptors and the performance of the particular delivery. Therecan be many rows of data, e.g. thousands of rows of data. Apattern-learning algorithm can be employed to process theconcatenations, i.e., rows of data, to identify which features of thedelivery seem to provide the best performance. The learned model ofthese features provides a performance predictor. Performance predictorscan be thought of as a set of rules that establish a relationshipbetween the features of the delivery and the actual value of the KPI.This can be performed by using supervised machine learning. Supervisedmachine learning is the machine learning task of inferring a functionfrom labeled training data. The training data consist of a set oftraining examples. In supervised learning, each sample is a pairconsisting of an input object, typically a vector (such as aconcatenation), and a desired output value. A supervised learningalgorithm analyzes the training data and produces an inferred function(in this case a predictor). An optimal scenario will allow for thealgorithm to correctly determine the class labels for unseen instanceswhich can be used for mapping new examples. A predictor is produced foreach KPI and each target dimension. The predictor predicts how therespective delivery will perform. A predictor is produced for each KPIand each target dimension. The predictor predicts how the respectivedelivery will perform.

The testing module then processes test content for the delivery that isto be employed by the marketer. The testing module receives contentassociated with the delivery, such as HTML files associated with thedelivery, along with input provided by the marketer such as KPIs and thetarget dimension to be tested. From this, the testing module extractsmeta-descriptors at the delivery level and processes themeta-descriptors along with information associated with how this contentperformed, e.g., according to a quantification scheme, to produce aconcatenation that describes meta-descriptors and content performancefor each delivery. The testing module then uses the predictor from thetraining module to output a predicted KPI value. That is, the set ofrules (i.e. the predictor), developed by the training module, whenapplied to the concatenation of meta-descriptors and content performancefor each delivery, produces a predicted KPI value for the individualdelivery.

A similar approach can be utilized at the offer level and themeta-descriptor level. The described approaches can be applied topredict the performance of a delivery, offer, and/or meta-descriptors atdesign time, before the delivery is actually produced and sent out toits consumers.

Various workflows can be utilized to enrich offers and deliveries withrelevant tags to help marketers create engaging content.

For example, content of a set of deliveries and meta-descriptorsextracted from the set of deliveries can be received. The content caninclude sets of HTML files associated with the delivery. Marketer inputin the form of KPI and target dimension can also be received. A semanticdescription of the meta-descriptors is created. So, for example, asemantic description can include “human smiling” for a meta-descriptorthat identifies a human face. Next, weights can be assigned to themeta-descriptors according to the performance prediction describedabove. That is, when the predictors learn which meta-descriptors of aparticular delivery lead to better KPI values, weights can be assignedaccordingly. Thus, better performing meta-descriptors will receivehigher weights. With the weights and the semantic descriptions havingbeen created, tags associated with the deliveries can be enriched. Thetags are enriched by associating the weights and the semanticdescriptions with the individual respective deliveries or offers. Thiscreates a searchable collection of deliveries that can be searched bythe enriched tags. Now, when a marketer searches for a particular tag,e.g., “smiling faces”, deliveries or offers can be presented thatinclude smiling faces, in decreasing order of weights. In this manner,marketers can see which deliveries or offers performed better for aparticular searched tag. This can enable a marketer to select aparticular delivery, which performed better than other deliveries, touse in their marketing campaign.

In the following discussion, an example digital medium environment isfirst described that can employ the techniques described herein. Exampleimplementation details and procedures are then described which can beperformed in the example digital medium environment as well as otherenvironments. Consequently, performance of the example procedures is notlimited to the example environment and the example environment is notlimited to performance of the example procedures.

Example Digital Medium Environment

FIG. 1 is an illustration of a digital medium environment 100 in anexample implementation that is operable to employ techniques describedherein. As used herein, the term “digital medium environment” refers tothe various computing devices and resources that can be utilized toimplement the techniques described herein. The illustrated digitalmedium environment 100 includes a computing device 102 including aprocessing system 104 that includes one or more processing devices, oneor more computer-readable storage media 106, and various applications108 embodied on the computer-readable storage media 106 and operable viathe processing system 104 to implement corresponding functionalitydescribed herein.

In at least some implementations, applications 108 include or otherwisemake use of a delivery building module 109. In some implementations, thedelivery building module 109 is a standalone application. In otherimplementations, the delivery building module 109 is included as part ofanother application or system software such as a computing device'soperating system. The delivery building module 109 is configured toenable meta-descriptors to be defined, selected and used to characterizeoffers in a theme-agnostic manner which promotes indexing the offers,identifying related offers, and searching for offers, as described belowin more detail. Theme-agnosticism pertains to how an offer looks in aparticular delivery, irrespective of an offer's theme.

Offers and content within offers can be automatically processed toidentify relevant meta-descriptors. This can be done using hardware andsoftware that analyzes the offers and content for context such asimages, color, text, fonts, font sizes, layout, text-to-imagepercentage, and the like. Meta-descriptors can include, by way ofexample and not limitation, color meta-descriptors, font and stylemeta-descriptors, delivery layout meta-descriptors, humanmeta-descriptors, and product/offer/delivery theme meta-descriptors.

Color meta-descriptors can be developed by extracting informationassociated with hue and saturation of images that appear in an offer.Hue is determined by the dominant wavelength within the visible lightspectrum at which the energy output from a source is greatest.Saturation is determined by the excitation purity, and depends on theamount of white light mixed with the hue. Color meta-descriptors canalso be developed by extracting information on how colors are spatiallyarranged within an image. Typically, an image sensor, such as a machinevision camera, can be used to capture an image which can then beprocessed by image processing software to extract color data and performanalysis to develop information pertaining to the hue, saturation, andspatial arrangement of colors within an image.

Font and style meta-descriptors can be developed by analyzing offers toascertain the fonts that appear within the offer, the size of the fonts,and various styles (e.g., formatting and layout properties) associatedwith the offer. This can be done in an automated way using opticalcharacter recognition (OCR) analysis to extract text from an offer andthen using processing software to process the extracted text to identifyfont and style information.

Delivery layout meta-descriptors can be developed by analyzing thedelivery to ascertain how the delivery's content is laid out. This canbe done by analyzing a delivery, via analysis software, to ascertain howand where images, offers, banners, and other content are arranged.

Human meta-descriptors can be developed by using image processingsoftware to identify human images and attributes of human images thatappear within a particular offer. Image processing software can includefacial detection and facial recognition software that can recognizehuman faces and can ascertain properties or characteristics of the humanfaces that appear within an offer. Such software can utilize facialrecognition algorithms to identify facial features by extractinglandmarks, or features, from an image of a subject's face. For example,the algorithm may analyze the relative position, size, and/or shape ofthe eyes, nose, cheekbones, and jaw.

Product/offer/delivery theme meta-descriptors can be developed bydetermining the number of products in an offer, number of offers in adelivery, number of images in the offer or delivery, proportion ofspecial sales and discount to offers, and theme of the delivery.

Once the meta-descriptors have been developed, the delivery buildingmodule 109 associates the meta-descriptors with the offers or deliveriesby including the meta-descriptors as part of an offer's or delivery'smetadata which, in turn, helps in indexing and searching for offers ordeliveries, as described below in more detail. For example, the offersand deliveries can be automatically saved in a searchable database alongwith metadata, i.e. meta-descriptors, that are associated with eachoffer and delivery. These meta-descriptors can then form the basis of asearchable index to easily identify offers or deliveries that meetparticular search criteria entered by a marketer.

This constitutes an improvement over current approaches which use aprimarily manual approach in which content is tagged by a marketer. Themanual approach can be very subjective and can vary widely depending onthe individual applying the tags. This leads to a non-standardizedapproach which, in turn, is not easily scalable in a predictable way.The delivery building module 109 is also configured to automaticallygenerate rules for assigning offers to outbound deliveries, irrespectiveof the availability of a particular customer's personal information.This enables the system to easily deal with new customers for whomhistoric information has not yet been developed. The delivery buildingmodule 109 is also configured to utilize separate workflows for offerperformance analytics and prediction at three levels of details—theoverall delivery level (e.g., the email level), the offer level, and themeta-descriptor level. The delivery building module 109 utilizesdifferent novel workflows for providing a predictive modeling approach,to predict the performance of an offer with respect to a marketer'sKPIs. At each level, the predictive modeling approach uses a trainingmodule and a testing module.

For example, at the overall delivery level, the training moduleprocesses training content associated with the delivery, such as HTMLfiles associated with the delivery, along with input provided by amarketer such as key performance indicators (KPI) such as open rate,click rate, and the like, as well as target dimension. The trainingmodule extracts meta-descriptors at the delivery level and processes themeta-descriptors along with information associated with how this contentperformed to produce a concatenation of meta-descriptors and contentperformance for each delivery. Content performance can be quantifiedusing any suitable quantification scheme, e.g., on a scale between1-100, with 1 being the worst and 100 being the best. Each concatenationconsists of one row of data which includes the meta-descriptors and theperformance of the particular delivery. There can be many rows of data,e.g. thousands of rows of data. A pattern-learning algorithm can beemployed to process the concatenations, i.e., rows of data, to identifywhich features of the delivery seem to provide the best performance. Thelearned model of these good features provides a performance predictor.Performance predictors can be thought of as a set of rules thatestablish a relationship between the features of the delivery and theactual value of the KPI. This can be performed by using supervisedmachine learning. A predictor is produced for each KPI and each targetdimension. The predictor predicts how the respective delivery willperform.

The testing module then processes test content for the delivery that isto be employed by the marketer. The testing module receives contentassociated with the delivery, such as HTML files associated with thedelivery, along with input provided by the marketer such as KPIs and thetarget dimension to be tested. From this, the testing module extractsmeta-descriptors at the delivery level and processes themeta-descriptors along with information associated with how this contentperformed, e.g., according to a quantification scheme, to produce aconcatenation of meta-descriptors and content performance for eachdelivery. The testing module then uses the predictor from the trainingmodule to output a predicted KPI value. That is, the set of rules (i.e.the predictor), developed by the training module, when applied to theconcatenation of meta-descriptors and content performance for eachdelivery, produces a predicted KPI value for the individual delivery.

A similar approach can be utilized at the offer level and themeta-descriptor level. The described approaches can be applied topredict the performance of a delivery, offer, and/or meta-descriptors atdesign time, before the delivery is actually produced and sent out toits consumers.

Applications 108 may also include a web browser which is operable toaccess various kinds of web-based resources (e.g., content andservices). In at least some implementations, the applications includeone or more video players, such as Adobe® Flash® Player, a YouTube™-typeapplication, and the like. In at least some implementations, theapplications 108 represent a client-side component having integratedfunctionality operable to access web-based resources (e.g., anetwork-enabled application), browse the Internet, interact with onlineproviders, and so forth. Applications 108 further include an operatingsystem for the computing device 102 and other device applications.

The computing device 102 may be configured as any suitable type ofcomputing device. For example, the computing device may be configured asa desktop computer, a laptop computer, a mobile device (e.g., assuming ahandheld configuration such as a tablet or mobile phone), a tablet, acamera, and so forth. Thus, the computing device 102 may range from fullresource devices with substantial memory and processor resources (e.g.,personal computers, game consoles) to a low-resource device with limitedmemory and/or processing resources (e.g., mobile devices). Additionally,although a single computing device 102 is shown, the computing device102 may be representative of a plurality of different devices to performoperations “over the cloud” as further described in relation to FIG. 15.

The digital medium environment 100 further depicts one or more serviceproviders 112, configured to communicate with computing device 102 overa network 114, such as the Internet, to provide a “cloud-based”computing environment. Generally speaking, a service provider 112 isconfigured to make various resources 116 available over the network 114to clients. In some scenarios, users may sign up for accounts that areemployed to access corresponding resources, such as streaming video,from a provider. The provider may authenticate credentials of a user(e.g., username and password) before granting access to an account andcorresponding resources 116. Other resources 116 may be made freelyavailable, (e.g., without authentication or account-based access). Theresources 116 can include any suitable combination of services and/orcontent typically made available over a network by one or moreproviders. Some examples of services include, but are not limited to, anotification service (such as one that sends various types ofnotifications to applications 108 and delivery building module 109), acontent publisher service that distributes content, such as streamingvideos and the like, to various computing devices, an advertising serverservice that provides advertisements to be used in connection withdistributed content, a web development and management service, acollaboration service, a social networking service, a messaging service,and so forth. Content may include various combinations of assets, videocomprising part of an asset, advertisements, audio, multi-media streams,animations, images, web documents, web pages, applications, deviceapplications, and the like.

Various types of input devices and input instrumentalities can be usedto provide input to computing device 102. For example, the computingdevice can recognize input as being a mouse input, stylus input, touchinput, input provided through a natural user interface, and the like.Thus, the computing device can recognize multiple types of gesturesincluding touch gestures and gestures provided through a natural userinterface.

Having considered an example digital medium environment, consider nowtwo example offers that can be processed by the inventive embodimentsdescribed herein.

FIG. 2 illustrates two different offers that can appear in a delivery,generally at 200. A first of the offers is illustrated at 202 and asecond of the offers is illustrated at 204. First offer 202 is a puretextual offer that incorporates the use of various font sizes, includinga somewhat larger font size to convey “70% off”. Second offer 204 is amixed offer and includes both images and textual components. The primaryfocus of this offer is the human images, with less emphasis on thetextual components. The various embodiments described herein aredirected to predicting the performance of these different kinds ofoffers and deliveries and others, as well as enabling the offers to beassigned to various deliveries. As such, the embodiments described beloware directed to solving three different problems that face marketers—(1)how to assign offers to marketing deliveries, (2) how to score theperformance of content in marketing deliveries, and (3) how to usedelivery performance information, such as email performance information,to send more relevant offers to recipients.

The solution to the problem of assigning offers to marketing deliveries(item (1)) is achieved by breaking down emails or marketing messagecontent to a level of the features of its content and offers, so as tobetter organize and identify related offers. These features are definedby a novel collection of meta-descriptors described below. The solutionof how to score the performance of content (item (2)) is achieved byusing the features (meta-descriptors) created just above, to measure theperformance of the content, and thus predict the performance of a futuredelivery based on these features. This solution is far more than simplymeasuring whether a particular offer was clicked or otherwise selected.For example, the solution can measure what types, sizes, and colors ofimages were selected, to name just a few. The solution to the problem ofsending more relevant offers (item (3)) is achieved by using thefeatures (meta-descriptors) created above to identify recipientpreferences at a finer level, and thus suggest the right recipients foran offer or the collection of content in a delivery.

The various implementations thus improve upon the current state of theart by providing an improved system that provides a diagnostic orpredictive tool which actually affixes the estimated or predictedcontribution of a layout to conversion and, at the same time, considersa rich set of meta-descriptors in delivery performance. Conversionrefers to conversion rate. The various embodiments are targeted to helpmarketers who are sending deliveries, such as email deliveries. In thatcontext, conversion means the revenue generated from these emaildeliveries. Conversion can be calculated in various ways, such as numberof clicks, number of transactions, revenue from transactions, and so on.Because each of the tagged offers and assets in an email plays a role inthe performance of the email, through this we can establish how muchlayout of a delivery has contributed to certain key performanceindicator KPI performance. An example of how this can be done isprovided below in connection with FIGS. 11-14 and the related discussionbelow.

The improved system also enables ranking of the best performing layouts,color schemes, and/or themes. Thus, the improved system described hereinprovides a robust approach to analyzing visual features of marketingcommunications for predicting performance.

Example Delivery Building Module

FIG. 3 illustrates an example delivery building module 209 in accordancewith one or more embodiments. In the illustrated example, deliverybuilding module 209 includes a meta-descriptor module 302, an outbounddelivery performance prediction module 304, and an outbound deliveryoffer assignment module 306, each of which is separately described belowunder its own heading. The modules can be implemented in connection withany suitable hardware, software, firmware or combination thereof. In atleast some embodiments, the modules are implemented in software thatresides on some type of non-transitory computer-readable media.

Meta-Descriptor Module

In accordance with one or more embodiments, the delivery building module209, by way of the meta-descriptor module 302, processes multiple offersin an automated manner to extract meta-descriptors that aretheme-agnostic. The meta-descriptor module is configured to extractmultiple different meta-descriptors by using multiple differentrespective modules to extract individual respective meta-datadescriptors. In the illustrated and described example themeta-descriptors include color meta-descriptors, font and stylemeta-descriptors, delivery layout meta-descriptors, humanmeta-descriptors, and product/offer/delivery theme meta-descriptors.Accordingly, the delivery building module can utilize a colormeta-descriptor module, a font and style meta-descriptor module, adelivery layout meta-descriptor module, a human meta-descriptor module,and a product/offer/delivery theme meta-descriptor module to extractindividual respective meta-descriptors, each of which is discussed belowunder its own heading.

Color Meta-Descriptors

Color plays an important role in determining performance of a delivery.While a sharp and vivid image can attract users to explore products, adull color theme may shy them away. Accordingly, in various embodiments,the meta-descriptor module 302 processes offers of a delivery to extractrelevant features to describe the color theme of the overall deliveryand its individual offers. This can include extracting informationassociated with hue and saturation of all images in an offer that aregreater than a particular size, and building a histogram for each offerthat includes the hue and saturation information. For example, aparticular delivery may contain a large number of images some of whichmay be quite small in size, such as icons and the like. The smallerimages are not processed to develop hue and saturation informationpartly because the smaller images typically play a negligible role inthe success of a campaign. Rather, only images meeting a certain sizethreshold are processed. The size threshold can be set by a marketer,via a suitably configured user interface, and can vary depending on themarketer's needs.

An image's hue and saturation can be extracted in any suitable way. Forexample, an image sensor, such as a machine vision camera, can be usedto capture an image which can then be processed by image processingsoftware to extract color data and perform analysis to developinformation pertaining to the hue and saturation that can be used tobuild a histogram for an offer.

In addition, the delivery is processed by an automated color layoutdescriptor to describe the spatial arrangement of colors. The colorlayout descriptor produces various descriptors, such as scalable colordescriptors that describe the color distribution in an image anddominant color descriptors which describe global as well as localspatial color distribution in an image. Any suitable color layoutdescriptor can be utilized, an example of which is described in Sikora,Thomas. “The MPEG-7 visual standard for content description—anoverview.” IEEE Transactions on Circuits and Systems for VideoTechnology, 11.6 (2001):696-702. To date, color meta-descriptors, suchas those described above, have not been employed in connection withmarketing messages to facilitate in delivery performance analysis,delivery performance prediction and offer selection as described in thisdocument. Accordingly, processing the delivery as described aboveprovides information on what kinds of colors are present in thedelivery's images and how the colors are arranged. This information canbe used to define color meta-descriptors.

Font and Style Meta-Descriptors

Textual descriptions within offers can be an important predictor ofdelivery performance. That is, textual descriptions that are more“eye-catching” tend to be more successful than textual descriptions thatare not eye-catching. In accordance with one or more embodiments,attributes of textual content of the delivery and its component offersare measured by the meta-descriptor module 302 through the use ofseveral descriptors.

A first of the descriptors pertains to the proportion of text to theimage at the offer level or at the delivery level. This can be derivedfrom a word histogram and number of images greater than a certain sizein the delivery. A word histogram is a frequency distribution of wordsin the offer or delivery. Here, we consider a minimum size to representcontent, as some images are of a few pixels in dimension. Accordingly,we wish to avoid small text and images.

A second of the descriptors pertains to the style, color, and sizedescriptions of the text in the image, at the offer or delivery level.This is an important feature because the effect of text might vary dueto its size and style, e.g., a bigger, easy-to-read font might lead to ahigher click rate. This can be ascertained by observing the effects of aparticular offer or delivery, e.g. click through rate, and correlatingthe effect with the descriptors of the offer or delivery. This can bedone in an automated fashion by a system that processes and correlatesKPIs with meta-descriptors. For example, meta-descriptors associatedwith larger fonts may be correlated with higher click-through rates. Inthe illustrated and described example, optical character recognition(OCR) techniques are utilized to extract text from the offers. OCRtechniques typically convert different types of documents, such asscanned paper documents, PDF files or images into editable andsearchable data. An example of such techniques are described in Smith,Ray. “An overview of the Tesseract OCR engine.” ICDAR. IEEE, 2007. Todate, font and style meta-descriptors, such as those described above,have not been employed in connection with marketing messages tofacilitate in delivery performance analysis, delivery performanceprediction and offer selection as described in this document.

Delivery Layout Meta-Descriptors

Delivery layout can be important in determining a delivery'sperformance. Delivery layout pertains to how various images are arrangedin an offer and, for example, how the image and textual description ofthe offer are placed inside the image. Different delivery layouts arepossible for the same set of offers, but the delivery performance may bedifferent depending upon the ease of browsing within the particulardelivery. So, for the same set of assets—such as images, offers andbanners, one can arrange the assets differently to achieve severalalternatives of what the layout of the final delivery could be.

In accordance with one or more embodiments, delivery layout is capturedby the meta-descriptor module 302 through the followingmeta-descriptors: (1) number and size of images and offers; and (2)spatial histograms describing the placement of offers and textualdescriptions in the delivery. This is done, in at least someembodiments, by using a grid-based approach which divides an offer intogrids and then describes how images are displayed inside the grid usingthe spatial histogram. For example, consider the layout of an emaildelivery. The text in a delivery can be placed at the top, middle orbottom. So we can represent the area within the delivery as a grid, andthen use an algorithm to decide which is the best position to choose fortext. Each of these placements would presumably have the potential tohave a different effect on performance. So, for example, statistics canbe maintained on the performance effect of a delivery for differentlayouts, e.g., delivery layouts with text appearing in the top centergrid cell have click-through rates that are 25% higher than deliverylayouts with text appearing in the bottom center grid cell. Further, forthe same content, to identify the content's different locations withinan email layout, an x-y coordinate system can be used to describe theplacement of the content within an email. For the same content atdifferent positions, performance information can be ascertained when thecontent is placed in the different positions within the email layout. Inpractice, this information can be developed for millions of emails andseveral variants, depending on how many times and image was used and inhow many different positions it was used. This historical informationcan then help to train a machine learning model, which identifies thatover all of the positions within an email, certain positions workedbetter than others. Depending on how the data is filtered or subsetted,this can provide a “best choice” for an overall recipient base or acertain segment, such as a certain age group.

To date, delivery layout meta-descriptors, such as those describedabove, have not been employed in connection with marketing messages tofacilitate in delivery performance analysis, delivery performanceprediction and offer selection as described in this document.

Human Meta-Descriptors

Human images are often used in offers to attract a viewer's attention.FIG. 1 illustrates but one example of the use of human images in anoffer. In accordance with one or more embodiments, certain usefulattributes of human images, such as faces, are extracted by themeta-descriptor module 302 to assess the images' impact on performance.Any suitable techniques can be utilized to extract human images. Onenon-limiting example is described in Zhu, Xiangxin, and Deva Ramanan.“Face detection, pose estimation, and landmark localization in the wild”IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.Meta-descriptors that can be utilized include, by way of example notlimitation, (1) the number of people in an offer image or delivery; and(2) various attributes of people appearing in an image. The attributescan be attributes of the human face such as, by way of example and notlimitation, male or female, smiling or not smiling, and the like. Todate, human meta-descriptors, such as those described above, have notbeen employed in connection with marketing messages to facilitate indelivery performance analysis, delivery performance prediction and offerselection as described in this document.

Product/Offer/Delivery Theme Meta-Descriptors

Meta-descriptors in this category pertain to the overall theme andproduct descriptions in a delivery. The meta-descriptors can include, byway of example and not limitation, (1) number of products in the offercontent; (2) number of offers; (3) number of images in the offer ordelivery; (4) proportion of special sales and discount to offers; and(5) theme of the delivery such as seasons, festivals, types of products,and the like.

As noted above, the meta-descriptors, as extracted by themeta-descriptor module 302, facilitate indexing offers for ease ofsearchability when building a marketing campaign. For example, once themeta-descriptors for an offer or delivery are extracted, themeta-descriptors can be saved as metadata of the offer or delivery. Thiscan then enable indexing and easy searching. Further, themeta-descriptors can assist in identifying related offers, as describedbelow in more detail.

Having considered examples of meta-descriptors in accordance with one ormore embodiments, consider now an approach which is directed topredicting content performance which, in turn, can be used to enhanceoffers through the use of the outbound delivery performance predictionmodule 304.

Outbound Delivery Performance Prediction Module

In one or more embodiments, three different workflows are utilized bythe outbound delivery performance prediction module 304 for predictingcontent performance. The workflows occur at the overall delivery level,offer level, and meta-descriptor level. Each individual level has, inturn, a training module and a testing module. For example, the deliverylevel includes a delivery training module 304 a and a delivery testingmodule 304 b. The offer level includes a clickable training module 304c, a clickable testing module 304 d, a non-clickable training module 304e, and a non-clickable testing module 304 f. The meta-descriptor levelincludes a meta-descriptor training module 304 g and a meta-descriptortesting module 304 h. Operation of these modules is described inrelation to FIGS. 4-7 below. The training module is designed to train aperformance prediction model which, when trained, can then be used topredict content performance based on information that is intended to beused by a marketer in a marketing campaign.

The training module pertains to a training process that is designed tomitigate the effects of not having historic customer information. Thetraining module does this by processing training content, such as HTMLfiles, various KPIs, and input target dimensions to create aconcatenation of meta-descriptors and performance of the trainingcontent. The concatenation consists of a row of data which includes themeta-descriptors and the performance of the particular delivery. Therecan be many rows of data, e.g. thousands of rows of data. Apattern-learning algorithm can be employed to process theconcatenations, i.e., rows of data, to identify which features of thedelivery seem to provide the best performance. The learned model ofthese good features provides a performance predictor. Performancepredictors can be thought of as a set of rules that establish arelationship between the features of the delivery and the actual valueof the KPI. This can be performed by using supervised machine learning.A predictor is produced for each KPI and each target dimension. Thepredictor predicts how the respective delivery will perform. Thus, thetraining module is designed to provide historic-type customerinformation that can then be utilized by the testing module to testactual deliveries that are going to be utilized by a marketer. That is,the testing module then processes test content for the delivery that isto be employed by the marketer. The testing module receives contentassociated with the delivery, such as HTML files associated with thedelivery, along with input provided by the marketer such as KPIs and thetarget dimension to be tested. From this, the testing module extractsmeta-descriptors at the delivery level and processes themeta-descriptors along with information associated with how this contentperformed, e.g., according to a quantification scheme, to produce aconcatenation of meta-descriptors and content performance for eachdelivery. The testing module then uses the predictor from the trainingmodule to output a predicted KPI value. That is, the set of rules (i.e.the predictor), developed by the training module, when applied to theconcatenation of meta-descriptors and content performance for eachdelivery, produces a predicted KPI value for the individual delivery.That is, the training modules are used to build the predictive modelwhich is then used by the marketers to predict performance at thevarious levels. Consider now the training and testing modules at theoverall delivery level.

Overall Delivery Level

The overall delivery level pertains to measuring which meta-descriptorswould be useful to aid the marketer for a particular dimension and keyperformance indicator (KPI) at the delivery level. The workflow includesa training module and a testing module and is described in connectionwith the flow diagram of FIG. 4.

FIG. 4 describes an example procedure 400 for predicting performance atthe overall delivery level in accordance with one or more embodiments.Aspects of the procedure may be implemented in hardware, firmware, orsoftware, or a combination thereof. The procedures are shown as a set ofblocks that specify operations performed by one or more devices and arenot necessarily limited to the orders shown for performing theoperations by the respective blocks. In at least some implementationsthe procedures may be performed in a digital medium environment by asuitably configured device, such as the example computing device 102 ofFIG. 1 that makes use of a delivery building module 109, such as thatdescribed above.

Training via the delivery training module 304 a, which may be conductedoff-line, outputs a predictor to forecast overall delivery campaignperformance. At test time, the marketer provides inputs, and thetraining module receives training delivery content, such as HTML contentand the KPI and target dimension he or she is interested in (block 402).The KPIs can include any suitable KPIs such as open rate, click rate,and the like. A target dimension can include any suitable type of targetdimension and constitutes the desired target of a particular deliverysuch as age group, gender, a segment of people, and the like. This canbe provided by way of a suitably configured user interface or dashboard.Specifically, in one or more embodiments, the overall delivery trainingmodule takes as input, HTML files of the training outbound deliveriesand a list of KPIs for which meta-descriptors are to be measured. Theseinputs are entered into the training module and concatenated features,i.e. meta-descriptors and a response for each delivery HTML, i.e. howthe content performed, are created (block 404). Content performance canbe quantified using any suitable quantification scheme, e.g., on a scalebetween 1-100, with 1 being the worst and 100 being the best. Eachconcatenation consists of one row of data which includes themeta-descriptors and the performance of the particular delivery. Therecan be many rows of data, e.g. thousands of rows of data. The model thenlearns from this historic data, in order to predict the performance ofany new offer or email based on information about its contents.

Next, a performance predictor is constructed using the concatenatedfeatures and each of the KPIs as a label (block 406). A KPI or a keyperformance indicator is what the marketer is looking to optimize, suchas having maximum clicks, maximum number of transactions, or maximumrevenue generated from the delivery or offer. So the prediction modelwants to predict these KPIs from the meta-descriptors that have beenextracted from the data. A pattern-learning algorithm can be employed toprocess the concatenations, i.e., rows of data, to identify whichfeatures of the delivery seem to provide the best performance. Thelearned model of these good features provides a performance predictor.Performance predictors can be thought of as a set of rules thatestablish a relationship between the features of the delivery and theactual value of the KPI. This can be performed by using supervisedmachine learning.

Finally, predictors for each KPI and each target dimension are output(block 408). If a target dimension, such as an age rule of targeting18-25 year olds, is used in the training data, then the prediction thatis produced will describe what would work best for that targetdimension. Collectively then, predictions for different targetdimensions can be produced and can be used to ascertain which offers ordeliveries will perform best, i.e., for which target dimension, andrecommend to the marketer to send the offer or delivery to those people.

Testing via the delivery testing module 304 b can then be used to test amarketer's actual delivery content, such as HTML and the KPI and targetdimension that the marketer is interested in. In this example, input isreceived in the form of content, such as HTML files of the test outbounddelivery and KPI and target dimension to be tested (block 410). Fromthese inputs, concatenated features of meta-descriptors and responsesfor the test outbound delivery HTML, i.e. the content performance, arecreated (block 412). As above, content performance can be quantified inany suitable way. Next, a predicted KPI for the target dimension isoutput using the corresponding predictor from the training module (block414). That is, testing module uses the predictor from the trainingmodule to output a predicted KPI value. Specifically, the set of rules(i.e. the predictor), developed by the training module, when applied tothe concatenation of meta-descriptors and content performance for eachdelivery, produces a predicted KPI value for the individual delivery.

The predicted KPI for the target dimension provides the marketer with aperformance prediction relative to the overall delivery. In someembodiments, this can be shown as a dashboard to marketers so that themarketers may see how the email will perform. Consider now the trainingand testing modules at the offer level.

Offer Level

The marketer is typically also interested in component-level performanceof deliveries, such as clickable offers, as well as non-clickablecomponents such as background images and the like. Workflows can beutilized to measure the performance of both types of components usingseparate workflows for the clickable and non-clickable componentsbecause both components can affect performance differently. Clickablecomponents have a user input, whereas non-clickable components do nothave a user input. Because of this, for non-clickable components, a KPIestimation is performed to ascertain which kinds of KPIs are to beassigned to the non-clickable components. Having separate workflows forthese two different types of components facilitates this process. In theillustrated and described embodiment, each workflow has a trainingmodule and a testing module. This level measures how a particular offerwill perform.

FIG. 5 describes an example procedure 500 for predicting performance atthe clickable component level of the offer level in accordance with oneor more embodiments. Aspects of the procedure may be implemented inhardware, firmware, or software, or a combination thereof. Theprocedures are shown as a set of blocks that specify operationsperformed by one or more devices and are not necessarily limited to theorders shown for performing the operations by the respective blocks. Inat least some implementations the procedures may be performed in adigital medium environment by a suitably configured device, such as theexample computing device 102 of FIG. 1 that makes use of a deliverybuilding module 109, such as that described above.

Training via the clickable training module 304 c for the clickablecomponents outputs a predictor for each KPI and target dimension.Processing at this level is essentially the same as with the overalldelivery level, except at a finer level of granularity. Now, instead ofa training module to predict the KPI of the entire delivery, i.e. anemail, the process focuses on individual constituents such as clickableoffers. In the illustrated example, the training module receives, asinput, content such as HTML files of a training set comprisingdeliveries used for training the model, and a list of KPIs such as openrate, and the like, for each offer within the delivery, and targetdimensions (block 502). Next, features of meta-descriptors and responsesfor each offer from the delivery HTML, i.e. content performance, areextracted (block 504). Content performance can be quantified using anysuitable quantification scheme, e.g., on a scale between 1-100, with 1being the worst and 100 being the best. Each concatenation consists ofone row of data which includes the meta-descriptors and the performanceof the particular delivery. There can be many rows of data, e.g.thousands of rows of data. Next, a performance predictor is constructedusing features of the offers and each of the marketer's KPIs and targetdimensions as a label (block 506). A pattern-learning algorithm can beemployed to process the concatenations, i.e., rows of data, to identifywhich features of the offers seem to provide the best performance. Thelearned model of these good features provides a performance predictor.Performance predictors can be thought of as a set of rules thatestablish a relationship between the features of the offer and theactual value of the KPI. This can be performed by using supervisedmachine learning. Finally, predictors for each KPI and target dimensionare output (block 508).

Testing via the clickable testing module 304 d can then be used to testthe marketer's clickable offer. To do so, content such as HTML files ofa test set comprising deliveries to be tested are received as input. Inaddition, the marketer's KPI and target dimension to be tested arereceived (block 510). The testing module then creates concatenatedfeatures of meta-descriptors and responses, content performance, for theoffer (block 512). The predicted KPI for the target dimension using thecorresponding predictor from the training module is then output (block514). That is, the testing module uses the predictor from the trainingmodule to output a predicted KPI value. Specifically, the set of rules(i.e. the predictor), developed by the training module, when applied tothe concatenation of meta-descriptors and content performance for eachoffer, produces a predicted KPI value for the individual delivery.

The predicted KPI value provides the marketer with a prediction of howclickable components of an offer may perform. That is, an individual KPImay have a range of values that indicate the degree of performance ofthe clickable component. Clickable components with lower values mayindicate poorer performance, while clickable components with highervalues may indicate higher levels of performance. Hence, by virtue ofthe KPIs value, a marketer can ascertain performance as betweenclickable components.

FIG. 6 describes an example procedure 600 for predicting performance atthe non-clickable component level of the offer level in accordance withone or more embodiments. Aspects of the procedure may be implemented inhardware, firmware, or software, or a combination thereof. Theprocedures are shown as a set of blocks that specify operationsperformed by one or more devices and are not necessarily limited to theorders shown for performing the operations by the respective blocks. Inat least some implementations the procedures may be performed in adigital medium environment by a suitably configured device, such as theexample computing device 102 of FIG. 1 that makes use of a deliverybuilding module 109, such as that described above.

Training via the non-clickable training module 304 e for thenon-clickable component provides predictors for each KPI and targetdimension of a non-clickable component. To do so, the non-clickablecomponent training module receives, as input, content such as HTML filesof a training set comprising deliveries used for training the model, aswell as a list of KPIs such as open rate or click rate for the overallemail, as well as each component. Input is also received in the form ofthe target dimension (block 602). From this, concatenated features ofmeta-descriptors and responses, i.e. content performance, for eachdelivery component, e.g., non-clickable components, are created (block604). Content performance can be quantified using any suitablequantification scheme, e.g., on a scale between 1-100, with 1 being theworst and 100 being the best. Each concatenation consists of one row ofdata which includes the meta-descriptors and the performance of theparticular delivery. There can be many rows of data, e.g. thousands ofrows of data. Next, a performance predictor is constructed using theconcatenated features and each of the KPIs of each component as a label(block 606). Here, the KPI of the overall delivery is used as the KPI ofthe non-clickable component. A pattern-learning algorithm can beemployed to process the concatenations, i.e., rows of data, to identifywhich features of the non-clickable components seem to provide the bestperformance. The learned model of these good features provides aperformance predictor. Performance predictors can be thought of as a setof rules that establish a relationship between the features of thenon-clickable components and the actual value of the KPI. This can beperformed by using supervised machine learning. From this, predictorsfor each of the KPIs and target dimension are output (block 608).

Testing via the non-clickable testing module 304 f for the non-clickablecomponent receives, as input, content such as HTML files of a test setcomprising deliveries used for testing the model. In addition, input inthe form of the marketer's input of KPI and target dimension is alsoreceived (block 610). From this, concatenated features ofmeta-descriptors and responses, i.e. content performance, for thenon-clickable components is created (block 612) and a predicted KPI forthe target dimension using the corresponding predictor from the trainingmodule is output (block 614). That is, the testing module uses thepredictor from the training module to output a predicted KPI value.Specifically, the set of rules (i.e. the predictor), developed by thetraining module, when applied to the concatenation of meta-descriptorsand content performance for each non-clickable component, produces apredicted KPI value. This workflow essentially learns how to predict theKPI's, for non-clickable components, like a background image. Itprovides a model that can predict the performance for any newnon-clickable component, given its meta-descriptors, based on thishistoric performance information. Consider now training and testingmodules at the meta-descriptor level.

Meta-Descriptor Level

The marketer may, in many instances, be interested in testing whethercertain meta-descriptors have good performance or not. For example, themarketer may be interested in whether a smiling human face increases theKPI or whether colorful images attract more customers. In theillustrated and described embodiment, two workflows can be utilized in amanner similar to that described above, i.e., a meta-descriptor trainingmodule and a meta-descriptor testing module.

FIG. 7 describes an example procedure 700 for predicting performance atthe meta-descriptor level of the offer level in accordance with one ormore embodiments. Aspects of the procedure may be implemented inhardware, firmware, or software, or a combination thereof. Theprocedures are shown as a set of blocks that specify operationsperformed by one or more devices and are not necessarily limited to theorders shown for performing the operations by the respective blocks. Inat least some implementations the procedures may be performed in adigital medium environment by a suitably configured device, such as theexample computing device 102 of FIG. 1 that makes use of a deliverybuilding module 109, such as that described above.

The meta-descriptor training module 304 g receives, as input, contentsuch as HTML files of a training set of deliveries and meta-descriptorsto be tested. In addition, the input also includes a list of KPIs suchas open rate, click rate, and the like for the overall delivery, as wellas each component. In addition, a target dimension is also received asinput (block 702). From this, meta-descriptors from each overalldelivery HTML, as well as its components, are created (block 704).Meta-descriptors are extracted for the overall delivery, as well as itsdifferent components, such as clickable and non-clickable components. Aperformance predictor is constructed using the meta-descriptor featureand each of the KPIs as a label (block 706), using a supervised learningmachine as described above. Predictors for each KPI and target dimensionare then output (block 708).

The meta-descriptor testing module 304 h receives, as input, contentsuch as HTML files of a test set of deliveries and meta-descriptors tobe tested. In addition, input in the form of a marketer's KPI and targetdimension to be tested is also received (block 710). From this, ameta-descriptor feature for the test delivery HTML is created (block712), as described above. A predicted KPI is output for the targetdimension using the corresponding predictor from the training module(block 714), as described above. That is, the testing module uses thepredictor from the training module to output a predicted KPI value.Specifically, the set of rules (i.e. the predictor), developed by thetraining module, when applied to the meta-descriptor feature produces apredicted KPI value for the target dimension.

Having considered various aspects of outbound delivery performanceprediction, consider now an example outbound delivery offer assignmentworkflow that can utilize the information developed just above.

Outbound Delivery Offer Assignment Module

The above-described workflows can be utilized to assist the marketer incomposing an engaging campaign. That is, for example, when thepredictors described above learn which meta-descriptors of an email seemto lead to a better KPI, weights can be assigned to the meta-descriptorsaccordingly and used as tags. Marketers can then search for particulartags to find deliveries that are then presented in decreasing order ofweights. As an example, consider the following.

FIG. 8 describes an example procedure 800 for presenting deliveries to amarketer when the marketer searches for a particular tag in accordancewith one or more embodiments. Aspects of the procedure may beimplemented in hardware, firmware, or software, or a combinationthereof. The procedures are shown as a set of blocks that specifyoperations performed by one or more devices and are not necessarilylimited to the orders shown for performing the operations by therespective blocks. In at least some implementations the procedures maybe performed in a digital medium environment by a suitably configureddevice, such as the example computing device 102 of FIG. 1 that makesuse of a delivery building module 109, such as that described above.

In accordance with one or more embodiments, outbound delivery offerassignment module 306 (FIG. 3) receives, as input, content such as HTMLfiles of a set of deliveries and extracted meta-descriptors. The inputalso includes marketer input in the form of KPI and target dimensions(block 802). Next, a semantic description of the meta-descriptors iscreated (block 804). So, for example, a semantic description can include“human smiling” for a meta-descriptor that identifies a human face.Weights are then assigned according to the performance predictionworkflow (block 806) described above. That is, when the predictors learnwhich meta-descriptors of a particular delivery lead to better KPIvalues, weights can be assigned accordingly. Thus, better performingmeta-descriptors will receive higher weights. Weights can be assigned inany suitable way. In at least some embodiments weights can be assignedas described in Blum, A., Langley, P., “Selection of relevant featuresand samples and machine learning.” Artificial Intelligence 97 (1997),245-271. Next, tags for deliveries are enriched using the semanticdescription and the calculated weights (block 808). The tags areenriched by associating the weights and the semantic descriptions withthe individual respective deliveries. This creates a searchablecollection of deliveries that can be searched by the enriched tags. Forexample, a delivery can be tagged as containing smiling faces, and acorresponding weight can be attached as described earlier.

When a tag is searched by a marketer, deliveries are presented to themarketer in decreasing order of weights (block 810). For example, amarketer can search for different tags like “smiling face” or “red textimages”. The deliveries containing these tags would be presented tomarketer in decreasing order of performance, or in decreasing order oftheir contribution to the prediction model. This provides deliverieswhich can provide the maximum predicted KPI.

FIG. 9 describes an example procedure 900 for assigning offers todeliveries in accordance with one or more embodiments. Aspects of theprocedure may be implemented in hardware, firmware, or software, or acombination thereof. The procedures are shown as a set of blocks thatspecify operations performed by one or more devices and are notnecessarily limited to the orders shown for performing the operations bythe respective blocks. In at least some implementations the proceduresmay be performed in a digital medium environment by a suitablyconfigured device, such as the example computing device 102 of FIG. 1that makes use of a delivery building module 109, such as that describedabove.

In accordance with one or more embodiments, outbound delivery offerassignment module 306 (FIG. 3) receives as input, deliveries and offersthe marketer wishes to propose (block 902). The outbound delivery offerassignment module then generates features (e.g., meta-descriptors) ofthe deliveries (block 904). Tags for the deliveries are enriched usingsemantic descriptions and calculated weights, as described above (block906). Next, the current deliveries are compared against previousdeliveries and offers using a similarity computation against tags and/orfeatures (block 908). There are many methods to identify whichdeliveries or offers are similar to each other. In at least someembodiments, this is done by comparing the meta-descriptors of onedelivery to another. For example, a previous delivery that matched threeout of five meta-descriptors of the current delivery and had the samesemantic description as the current delivery will have a higher scorethan a previous delivery that matched one out of five meta-descriptorsof the current delivery and had a semantic description that did notmatch the current delivery. This enables marketers to browse throughprevious deliveries with similar tags and meta-descriptors. Marketerscan then look for the most similar, yet better performingoffers/meta-descriptors like layout and then incorporate them in thecurrent delivery. Similar deliveries and offers can be presented to themarketer along with their historic performance (block 910). This assiststhe marketers in assigning similar related, high-performing offers intodeliveries.

Essentially, the process described just above enables a workflow topresent similar high-performing historic offers to marketers so that themarketer can compose the delivery accordingly. This uses the new tagsthat were provided just above, compared against the tags of the requireddelivery. A similarity score is computed to identify and list relatedoffers, sorted according to their similarity and historic performance.The marketer can then choose the appropriate offer, based on itssimilarity and historic performance.

FIG. 10 describes an example procedure 1000 for assigning offers todeliveries in accordance with one or more embodiments, in whichassignments are made specific to recipients. Aspects of the proceduremay be implemented in hardware, firmware, or software, or a combinationthereof. The procedures are shown as a set of blocks that specifyoperations performed by one or more devices and are not necessarilylimited to the orders shown for performing the operations by therespective blocks. In at least some implementations the procedures maybe performed in a digital medium environment by a suitably configureddevice, such as the example computing device 102 of FIG. 1 that makesuse of a delivery building module 109, such as that described above.

Content of a set of deliveries, such as HTML files and extractedmeta-descriptors are received as input to the outbound delivery offerassignment module (block 1002). In addition, features (e.g.,meta-descriptors) of the deliveries are generated (block 1004) and KPIand target dimensions are received from the marketer (block 1006).Delivery and offer performance for aggregate consumers is predictedusing a trained model to predict the click rate (block 1008), asdescribed above. Delivery and offer performance for an individualrecipient is predicted using a trained model based on the user's historyalone (block 1010). Offers and deliveries are then arranged indecreasing order of predicted performance and presented via a suitableuser interface (block 1012). Deliveries and offers can then be assignedto individual recipients.

Experimental Setup

Consider the following experimental set up which employs the followingworkflow. The delivery building module receives input HTML files ofdeliveries and from these files creates a histogram of meta-descriptorsand responses, i.e. performance. The histogram is the input set offeatures or independent variables which is used by the model to identifypatterns and relationships between independent variables and the KPI.For example, emails with red font color always get a high click rate, isone kind of inference that this automatic model can make.Meta-descriptor vectors of the deliveries are constructed. Ameta-descriptor vector is a list of meta-descriptors for the entiredelivery. Next, the KPIs are identified and the model is trained againstobserved behavior or other ground truths. This can be done usingsupervised machine learning as described above. The model is thenapplied to unseen data to derive the expected behavior and a sorted listof meta-descriptors is created based on their contribution toward themarketer's KPI. As an example, consider the following datasetdescription.

Dataset Description

A dataset of 7.9 million deliveries was developed, sent to 3.8 millionrecipients over a period of 1 week, through 7 email campaigns, which arerepresented in the table illustrated in FIG. 11. The table in FIG. 11presents some of the raw frequencies for some of the meta-descriptors,at the level of the delivery. The campaign code is shown in the far leftcolumn. In this example, the KPI that is sought to be predicted is the“click-to-open rate.” The relevant meta-descriptors are shown startingwith column designated “Number of images”. Each meta-descriptor columncontains the raw frequencies for each campaign. So, for campaign“DM114668” the number of images was 14, and so on.

FIG. 12 illustrates the raw frequencies for each of the content piecesin delivery “DM114668”, where individual offers have an average clickrate of 0.108 and a standard deviation of 0.113. Finally, a qualitative,pairwise comparison of two email deliveries is conducted to ascertainwhether the difference in meta-descriptors is worth exploring. Forexample, FIG. 13 illustrates the email content of deliveries “DM114668”and “DM115442.” Delivery “DM114668” has an average click-to-open rate of9.2 clicks per 100 opens, and “DM115442” has an average click-to-openrate of 13.6 clicks per 100 opens. After normalizing the raw frequenciesfor the proportionate presence of meta-descriptors, the histogramillustrated in FIG. 14 was obtained. This shows the contrast in themeta-descriptors of the two deliveries. Accordingly, the marketer wouldknow, from the prediction model, the weights of each of themeta-descriptors, or how much the meta-descriptors contribute to theKPI. If the marketer is choosing between two meta-descriptors, he or shewould naturally want to choose offers/deliveries with thosemeta-descriptors which are positively and strongly correlated with theKPI, according to the model. This illustrates which meta-descriptorswere found to be more important than other meta-descriptors. So forsimilar campaigns promoting similar products, the difference in theimportance of the meta-descriptors can be ascertained.

The tables of FIGS. 11 and 12 highlight the noticeable difference inmeta-descriptors, hinting that it could be worth exploring therelationship between meta-descriptors and delivery performance.

Having considered the above implementations, consider now an examplesystem and device that can be utilized to practice the inventiveprinciples described herein.

Example System and Device

FIG. 15 illustrates an example system generally at 1500 that includes anexample computing device 1502 that is representative of one or morecomputing systems and/or devices that may implement the varioustechniques described herein. This is illustrated through inclusion ofthe applications 208 and, in particular, delivery building module 209,which operates as described above. The computing device 1502 may be, forexample, a server of a service provider, a device associated with aclient (e.g., a client device), an on-chip system, and/or any othersuitable computing device or computing system.

The example computing device 1502 is illustrated as including aprocessing system 1504, one or more computer-readable media 1506, andone or more I/O interface 1508 that are communicatively coupled, one toanother. Although not shown, the computing device 1502 may furtherinclude a system bus or other data and command transfer system thatcouples the various components, one to another. A system bus can includeany one or combination of different bus structures, such as a memory busor memory controller, a peripheral bus, a universal serial bus, and/or aprocessor or local bus that utilizes any of a variety of busarchitectures. A variety of other examples are also contemplated, suchas control and data lines.

The processing system 1504 is representative of functionality to performone or more operations using hardware. Accordingly, the processingsystem 1504 is illustrated as including hardware elements 1510 that maybe configured as processors, functional blocks, and so forth. This mayinclude implementation in hardware as an application specific integratedcircuit or other logic device formed using one or more semiconductors.The hardware elements 1510 are not limited by the materials from whichthey are formed or the processing mechanisms employed therein. Forexample, processors may be comprised of semiconductor(s) and/ortransistors (e.g., electronic integrated circuits (ICs)). In such acontext, processor-executable instructions may beelectronically-executable instructions.

The computer-readable storage media 1506 is illustrated as includingmemory/storage 1512. The memory/storage 1512 represents memory/storagecapacity associated with one or more computer-readable media. Thememory/storage component 1512 may include volatile media (such as randomaccess memory (RAM)) and/or nonvolatile media (such as read only memory(ROM), Flash memory, optical disks, magnetic disks, and so forth). Thememory/storage component 1512 may include fixed media (e.g., RAM, ROM, afixed hard drive, and so on) as well as removable media (e.g., Flashmemory, a removable hard drive, an optical disc, and so forth). Thecomputer-readable media 1506 may be configured in a variety of otherways as further described below.

Input/output interface(s) 1508 are representative of functionality toallow a user to enter commands and information to computing device 1502,and also allow information to be presented to the user and/or othercomponents or devices using various input/output devices. Examples ofinput devices include a keyboard, a cursor control device (e.g., amouse), a microphone, a scanner, touch functionality (e.g., capacitiveor other sensors that are configured to detect physical touch), a camera(e.g., which may employ visible or non-visible wavelengths such asinfrared frequencies to recognize movement as gestures that do notinvolve touch), and so forth. Examples of output devices include adisplay device (e.g., a monitor or projector), speakers, a printer, anetwork card, tactile-response device, and so forth. Thus, the computingdevice 1502 may be configured in a variety of ways as further describedbelow to support user interaction.

Various techniques may be described herein in the general context ofsoftware, hardware elements, or program modules. Generally, such modulesinclude routines, programs, objects, elements, components, datastructures, and so forth that perform particular tasks or implementparticular abstract data types. The terms “module,” “functionality,” and“component” as used herein generally represent software, firmware,hardware, or a combination thereof. The features of the techniquesdescribed herein are platform-independent, meaning that the techniquesmay be implemented on a variety of commercial computing platforms havinga variety of processors.

An implementation of the described modules and techniques may be storedon or transmitted across some form of computer-readable media. Thecomputer-readable media may include a variety of media that may beaccessed by the computing device 1502. By way of example, and notlimitation, computer-readable media may include “computer-readablestorage media” and “computer-readable signal media.”

“Computer-readable storage media” refers to media and/or devices thatenable persistent and/or non-transitory storage of information incontrast to mere signal transmission, carrier waves, or signals per se.Thus, computer-readable storage media does not include signals per se orsignal bearing media. The computer-readable storage media includeshardware such as volatile and non-volatile, removable and non-removablemedia and/or storage devices implemented in a method or technologysuitable for storage of information such as computer readableinstructions, data structures, program modules, logic elements/circuits,or other data. Examples of computer-readable storage media may include,but are not limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other opticalstorage, hard disks, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or other storage device,tangible media, or article of manufacture suitable to store the desiredinformation and which may be accessed by a computer.

“Computer-readable signal media” refers to a signal-bearing medium thatis configured to transmit instructions to the hardware of the computingdevice 1502, such as via a network. Signal media typically may embodycomputer readable instructions, data structures, program modules, orother data in a modulated data signal, such as carrier waves, datasignals, or other transport mechanism. Signal media also include anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media include wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared, and other wireless media.

As previously described, hardware elements 1510 and computer-readablemedia 806 are representative of modules, programmable device logicand/or fixed device logic implemented in a hardware form that may beemployed in some implementations to implement at least some aspects ofthe techniques described herein, such as to perform one or moreinstructions. Hardware may include components of an integrated circuitor on-chip system, an application-specific integrated circuit (ASIC), afield-programmable gate array (FPGA), a complex programmable logicdevice (CPLD), and other implementations in silicon or other hardware.In this context, hardware may operate as a processing device thatperforms program tasks defined by instructions and/or logic embodied bythe hardware as well as a hardware utilized to store instructions forexecution, e.g., the computer-readable storage media describedpreviously.

Combinations of the foregoing may also be employed to implement varioustechniques described herein. Accordingly, software, hardware, orexecutable modules may be implemented as one or more instructions and/orlogic embodied on some form of computer-readable storage media and/or byone or more hardware elements 1510. The computing device 1502 may beconfigured to implement particular instructions and/or functionscorresponding to the software and/or hardware modules. Accordingly,implementation of a module that is executable by the computing device1502 as software may be achieved at least partially in hardware, e.g.,through use of computer-readable storage media and/or hardware elements1510 of the processing system 1504. The instructions and/or functionsmay be executable/operable by one or more articles of manufacture (forexample, one or more computing devices 1502 and/or processing systems1504) to implement techniques, modules, and examples described herein.

The techniques described herein may be supported by variousconfigurations of the computing device 1502 and are not limited to thespecific examples of the techniques described herein. This functionalitymay also be implemented all or in part through use of a distributedsystem, such as over a “cloud” 1514 via a platform 1516 as describedbelow.

The cloud 1514 includes and/or is representative of a platform 1516 forresources 818. The platform 1516 abstracts underlying functionality ofhardware (e.g., servers) and software resources of the cloud 1514. Theresources 1518 may include applications and/or data that can be utilizedwhile computer processing is executed on servers that are remote fromthe computing device 1502. Resources 1518 can also include servicesprovided over the Internet and/or through a subscriber network, such asa cellular or Wi-Fi network.

The platform 1516 may abstract resources and functions to connect thecomputing device 1502 with other computing devices. The platform 1516may also serve to abstract scaling of resources to provide acorresponding level of scale to encountered demand for the resources1518 that are implemented via the platform 1516. Accordingly, in aninterconnected device implementation, implementation of functionalitydescribed herein may be distributed throughout the system 1500. Forexample, the functionality may be implemented in part on the computingdevice 1502 as well as via the platform 1516 that abstracts thefunctionality of the cloud 1514.

CONCLUSION

Various embodiments are directed to assigning offers to marketingdeliveries utilizing new features to describe offers in the marketingdeliveries. Marketing deliveries can be described at a finer level tothus enhance the effectiveness of building and conducting marketingcampaigns. The approaches facilitate matching content to recipients,predicting content performance, and measuring content performance afterdispatching a marketing delivery.

Although the invention has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the invention defined in the appended claims is not necessarilylimited to the specific features or acts described. Rather, the specificfeatures and acts are disclosed as example forms of implementing theclaimed invention.

What is claimed is:
 1. One or more computer-readable storage mediacomprising instructions that are stored thereon that implement a deliverbuilding module configured to enable electronic deliveries to be builtas part of the marketing campaign, to include at least one offer as partof the marketing campaign, the delivery building module comprising: ameta-descriptor module configured to process offers of a delivery toextract meta-descriptors to facilitate indexing and searching of theoffers to include in the marketing campaign, the meta-descriptor moduleconfigured to extract multiple different meta-descriptors by usingmultiple different respective modules to extract individual respectivemeta-data descriptors; an outbound delivery performance predictionmodule configured to predict content performance of content as part ofthe marketing campaign by employing: training modules that usesupervised machine learning to process training content to producepredictors that predict how a delivery will perform, and testing modulesthat use the predictors to process test content to produce predicted keyperformance indicator (KPI) values for the test content; and an outbounddelivery offer assignment module configured to compose the marketingcampaign by processing content of a set of deliveries, key performanceindicators, and target dimensions to enable searching and presentationof the deliveries for selection by a marketer.
 2. The one or morecomputer-readable storage media as described in claim 1, wherein saidmeta-descriptors include at least one of color meta-descriptors, fontand style meta-descriptors, delivery layout meta-descriptors, humanmeta-descriptors, or theme-based meta-descriptors.
 3. The one or morecomputer-readable storage media as described in claim 2, wherein themeta-descriptor module is configured to use a color meta-descriptor toextract color meta-descriptors associated with hue and saturation ofimages within an offer, and layout descriptors that describe spatialarrangement of colors within the images.
 4. The one or morecomputer-readable storage media as described in claim 2, wherein thefont and style meta-descriptors include at least one of: a descriptorthat pertains to a proportion of text to an image in an offer; adescriptor that pertains to the style, color, and size descriptions oftext in an image; a delivery layout descriptor that pertains to howvarious images are arranged in an offer including number and size ofimages in an offer, and placement of offers and textual descriptions ina delivery.
 5. The one or more computer-readable storage media asdescribed in claim 2, wherein the human meta-descriptors include atleast one of: a number of people in an offer image or delivery; andattributes of people appearing in an image or delivery.
 6. The one ormore computer-readable storage media as described in claim 2, whereinthe theme-based meta-descriptors include at least one of: number ofproducts in content of an offer; number of offers; number of images inan offer or delivery; proportion of special sales and discount offers;or theme of a delivery.
 7. The one or more computer-readable storagemedia as described in claim 1, wherein the outbound delivery performanceprediction module is configured to predict content performance at adelivery level.
 8. The one or more computer-readable storage media asdescribed in claim 1, wherein the outbound delivery performanceprediction module is configured to predict content performance at anoffer level.
 9. The one or more computer-readable storage media asdescribed in claim 1, wherein the outbound delivery performanceprediction module configured to predict content performance at ameta-descriptor level.
 10. In a digital medium environment including acomputing device configured to enable a marketing campaign to be builtwhich include electronic deliveries that contain at least one offer, animproved performance prediction method, the method comprising:receiving, with the computing device, content that is to comprise partof deliveries of a marketing campaign; receiving, with the computingdevice, at least one key performance indicator and an input targetdimension for the marketing campaign; creating, with the computingdevice, concatenations of meta-descriptors associated with the contentand performance for each delivery; and outputting, with the computingdevice, a predicted key performance indicator for the input targetdimension using a corresponding predictor from a training module that isconfigured to use supervised machine learning to produce predictors. 11.A method as described in claim 10, wherein said receiving contentcomprises receiving HTML files of a test outbound delivery.
 12. A methodas described in claim 10, wherein said receiving content comprisesreceiving HTML files associated with an offer that includes clickablecontent.
 13. A method as described in claim 10, wherein said receivingcontent comprises receiving HTML files associated with an offer thatincludes non-clickable content.
 14. A method as described in claim 10,wherein said receiving content comprises receiving HTML files associatedwith a delivery and associated meta-descriptors to be tested.
 15. Asystem implemented in a digital medium environment in which a marketingcampaign is built to include electronic deliveries employing at leastone offer as part of the marketing campaign, the system comprising: aprocessing system; at least one computer readable media storinginstructions, executable via the processing system, to performoperations comprising: receiving content of a set of deliveries andextracted meta-descriptors that are associated with the marketingcampaign; receiving at least one key performance indicator (KPI) andtarget dimensions associated with the marketing campaign; creating asemantic description of the meta-descriptors and assigning weights tothe meta-descriptors according to a performance prediction workflow, theweights being assigned as a function of KPI values produced by theperformance prediction workflow for a given target dimension; enrichingat least one tag for the set of deliveries using the semanticdescription and the calculated weights; and enabling said at least onetag to be searched and, responsive to a search of the at least one tag,presenting associated deliveries in an order based on the weights. 16.The system as described in claim 15, wherein said receiving contentcomprises receiving at least one HTML file of the set of deliveries. 17.The system as described in claim 15, wherein the meta-descriptorsinclude at least one of color meta-descriptors, font and stylemeta-descriptors, delivery layout meta-descriptors, humanmeta-descriptors, or theme-based meta-descriptors.
 18. The system asdescribed in claim 15 further comprising, assigning at least one offerto at least one delivery.
 19. The system as described in claim 15further comprising, assigning at least one offer to at least onedelivery, wherein said assigning at least one offer to at least onedelivery is performed by: receiving at least one offer and at least onedelivery; generating at least one meta-descriptor of the at least onedelivery; enriching tags for the at least one delivery using semanticdescriptions of the at least one meta-descriptor and weights assigned tothe at least one meta-descriptor as a function of KPI values produced bythe performance prediction workflow for a given target dimension;comparing current deliveries against previous deliveries and offersusing a similarity computation; and based on said comparing, causingpresentation of similar deliveries and offers.
 20. The system asdescribed in claim 19, wherein said causing comprises presentinghistoric performance of the presented similar deliveries and offers.